A toolkit to generate 3D mesh model / video / nerf instance / multiveiw images of colourful 3D objects by text and image prompts input. Edited by Shengyu Meng (Simon)
Wellcome to share the outputs in social media with tag #dreamfields3D 😀
Beta v0.65 :add options for apply image prompts only in assigned direction to avoid overfitting; could skip saving depth maps in training and testing; output sequence images in validation process. (2022-10-03)
Beta v0.60 : apply random fovy (view angle) in training; update image augmentation before feed into CLIP; improve training stability and performance. (2022-09-25)
Dreamfields-3D is modified from ashawkey's dreamfields-torch, and its forks of IV Pravdin and Pollinations.AI, released under MIT License.
The main improvements in this notebook & repository were done by Shengyu Meng (Simon), including allow visualizing training process in colab, export 360° video and 3D mesh model with vertex colour, and more arguments.This code is released under MIT License
The original dreamfields was issued under Apache-2.0 license, proposed by Jain, Ajay and Mildenhall, Ben and Barron, Jonathan T. and Abbeel, Pieter and Poole, Ben in their paper, Zero-Shot Text-Guided Object Generation with Dream Fields published on CVPR 2022. The main different of dreamfields-torch compared with original dreamfields is, dreamfields-torch applied the torch version of instant-ngp to replace the original NeRF as backend, and re-write most of the codes.
The technical bases of original dreamfields are NeRF: Neural Radiance Fields, released under MIT License, proposed by Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng in their paper NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis published on ECCV 2020, and CLIP: Connecting Text and Images model developed by OpenAI, released under MIT License.
#@title ### Check GPU
!nvidia-smi
Tue Mar 28 04:08:42 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12 Driver Version: 525.85.12 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-SXM... Off | 00000000:00:04.0 Off | 0 |
| N/A 29C P0 44W / 400W | 0MiB / 40960MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| No running processes found |
+-----------------------------------------------------------------------------+
#@title ### Mount Google drive (otpional)
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
#@title ### pull the repostory & get working dir
import os
!git clone https://github.com/shengyu-meng/dreamfields-3D
%cd dreamfields-3D
DC_dir = os.getcwd()
Cloning into 'dreamfields-3D'... remote: Enumerating objects: 313, done. remote: Counting objects: 100% (158/158), done. remote: Compressing objects: 100% (80/80), done. remote: Total 313 (delta 103), reused 124 (delta 76), pack-reused 155 Receiving objects: 100% (313/313), 47.12 MiB | 23.31 MiB/s, done. Resolving deltas: 100% (165/165), done. /content/dreamfields-3D
#@title install dependencies
!pip install -r requirements.txt
!bash scripts/install_ext.sh
!bash scripts/install_PyMarchingCubes.sh
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Collecting git+https://github.com/openai/CLIP.git (from -r requirements.txt (line 24))
Cloning https://github.com/openai/CLIP.git to /tmp/pip-req-build-0loqucp2
Running command git clone --filter=blob:none --quiet https://github.com/openai/CLIP.git /tmp/pip-req-build-0loqucp2
Resolved https://github.com/openai/CLIP.git to commit a9b1bf5920416aaeaec965c25dd9e8f98c864f16
Preparing metadata (setup.py) ... done
Collecting torch-ema
Downloading torch_ema-0.3-py3-none-any.whl (5.5 kB)
Collecting ninja
Downloading ninja-1.11.1-py2.py3-none-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (145 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 146.0/146.0 KB 8.4 MB/s eta 0:00:00
Collecting trimesh
Downloading trimesh-3.21.2-py3-none-any.whl (680 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 680.5/680.5 KB 42.2 MB/s eta 0:00:00
Requirement already satisfied: opencv-python in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 4)) (4.7.0.72)
Collecting tensorboardX
Downloading tensorboardX-2.6-py2.py3-none-any.whl (114 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 114.5/114.5 KB 16.7 MB/s eta 0:00:00
Requirement already satisfied: torch in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 6)) (1.13.1+cu116)
Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 7)) (1.22.4)
Requirement already satisfied: pandas in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 8)) (1.4.4)
Requirement already satisfied: tqdm in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 9)) (4.65.0)
Requirement already satisfied: matplotlib in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 10)) (3.7.1)
Requirement already satisfied: rich in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 13)) (13.3.2)
Collecting dearpygui
Downloading dearpygui-1.9.0-cp39-cp39-manylinux1_x86_64.whl (2.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.8/2.8 MB 84.6 MB/s eta 0:00:00
Requirement already satisfied: packaging in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 16)) (23.0)
Requirement already satisfied: scipy in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 17)) (1.10.1)
Collecting lpips
Downloading lpips-0.1.4-py3-none-any.whl (53 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.8/53.8 KB 7.5 MB/s eta 0:00:00
Requirement already satisfied: imageio in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 19)) (2.25.1)
Requirement already satisfied: imageio-ffmpeg in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 20)) (0.4.8)
Collecting pymeshlab
Downloading pymeshlab-2022.2.post3-cp39-cp39-manylinux1_x86_64.whl (59.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.3/59.3 MB 21.0 MB/s eta 0:00:00
Collecting ipyplot
Downloading ipyplot-1.1.1-py3-none-any.whl (13 kB)
Requirement already satisfied: IPython in /usr/local/lib/python3.9/dist-packages (from -r requirements.txt (line 23)) (7.9.0)
Requirement already satisfied: protobuf<4,>=3.8.0 in /usr/local/lib/python3.9/dist-packages (from tensorboardX->-r requirements.txt (line 5)) (3.19.6)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from torch->-r requirements.txt (line 6)) (4.5.0)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.9/dist-packages (from pandas->-r requirements.txt (line 8)) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.9/dist-packages (from pandas->-r requirements.txt (line 8)) (2022.7.1)
Requirement already satisfied: importlib-resources>=3.2.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (5.12.0)
Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (3.0.9)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (0.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (1.4.4)
Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (1.0.7)
Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (8.4.0)
Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.9/dist-packages (from matplotlib->-r requirements.txt (line 10)) (4.39.2)
Requirement already satisfied: markdown-it-py<3.0.0,>=2.2.0 in /usr/local/lib/python3.9/dist-packages (from rich->-r requirements.txt (line 13)) (2.2.0)
Requirement already satisfied: pygments<3.0.0,>=2.13.0 in /usr/local/lib/python3.9/dist-packages (from rich->-r requirements.txt (line 13)) (2.14.0)
Requirement already satisfied: torchvision>=0.2.1 in /usr/local/lib/python3.9/dist-packages (from lpips->-r requirements.txt (line 18)) (0.14.1+cu116)
Collecting shortuuid
Downloading shortuuid-1.0.11-py3-none-any.whl (10 kB)
Requirement already satisfied: pickleshare in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (0.7.5)
Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (5.7.1)
Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (2.0.10)
Collecting jedi>=0.10
Downloading jedi-0.18.2-py2.py3-none-any.whl (1.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 80.1 MB/s eta 0:00:00
Requirement already satisfied: decorator in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (4.4.2)
Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (67.6.0)
Requirement already satisfied: backcall in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (0.2.0)
Requirement already satisfied: pexpect in /usr/local/lib/python3.9/dist-packages (from IPython->-r requirements.txt (line 23)) (4.8.0)
Collecting ftfy
Downloading ftfy-6.1.1-py3-none-any.whl (53 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.1/53.1 KB 7.6 MB/s eta 0:00:00
Requirement already satisfied: regex in /usr/local/lib/python3.9/dist-packages (from clip==1.0->-r requirements.txt (line 24)) (2022.10.31)
Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.9/dist-packages (from importlib-resources>=3.2.0->matplotlib->-r requirements.txt (line 10)) (3.15.0)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in /usr/local/lib/python3.9/dist-packages (from jedi>=0.10->IPython->-r requirements.txt (line 23)) (0.8.3)
Requirement already satisfied: mdurl~=0.1 in /usr/local/lib/python3.9/dist-packages (from markdown-it-py<3.0.0,>=2.2.0->rich->-r requirements.txt (line 13)) (0.1.2)
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.9/dist-packages (from prompt-toolkit<2.1.0,>=2.0.0->IPython->-r requirements.txt (line 23)) (1.16.0)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.9/dist-packages (from prompt-toolkit<2.1.0,>=2.0.0->IPython->-r requirements.txt (line 23)) (0.2.6)
Requirement already satisfied: requests in /usr/local/lib/python3.9/dist-packages (from torchvision>=0.2.1->lpips->-r requirements.txt (line 18)) (2.27.1)
Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.9/dist-packages (from pexpect->IPython->-r requirements.txt (line 23)) (0.7.0)
Requirement already satisfied: charset-normalizer~=2.0.0 in /usr/local/lib/python3.9/dist-packages (from requests->torchvision>=0.2.1->lpips->-r requirements.txt (line 18)) (2.0.12)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.9/dist-packages (from requests->torchvision>=0.2.1->lpips->-r requirements.txt (line 18)) (2022.12.7)
Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.9/dist-packages (from requests->torchvision>=0.2.1->lpips->-r requirements.txt (line 18)) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.9/dist-packages (from requests->torchvision>=0.2.1->lpips->-r requirements.txt (line 18)) (1.26.15)
Building wheels for collected packages: clip
Building wheel for clip (setup.py) ... done
Created wheel for clip: filename=clip-1.0-py3-none-any.whl size=1369398 sha256=482fb35cd3e903cdc3e3686d2ba151d2f2f0649a69eb1bea35574de71feed9b3
Stored in directory: /tmp/pip-ephem-wheel-cache-kz_pfq1b/wheels/c8/e4/e1/11374c111387672fc2068dfbe0d4b424cb9cdd1b2e184a71b5
Successfully built clip
Installing collected packages: ninja, trimesh, tensorboardX, shortuuid, pymeshlab, jedi, ftfy, dearpygui, torch-ema, lpips, ipyplot, clip
Successfully installed clip-1.0 dearpygui-1.9.0 ftfy-6.1.1 ipyplot-1.1.1 jedi-0.18.2 lpips-0.1.4 ninja-1.11.1 pymeshlab-2022.2.post3 shortuuid-1.0.11 tensorboardX-2.6 torch-ema-0.3 trimesh-3.21.2
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Processing /content/dreamfields-3D/raymarching
Preparing metadata (setup.py) ... done
Building wheels for collected packages: raymarching
Building wheel for raymarching (setup.py) ... done
Created wheel for raymarching: filename=raymarching-0.0.0-cp39-cp39-linux_x86_64.whl size=2812222 sha256=e96f61911824536f03f6a62ed18d36a9200ca445a8ef435c0efad2b2ae92cb83
Stored in directory: /tmp/pip-ephem-wheel-cache-_5cnakc6/wheels/39/89/49/3e28339c4b4f791a9aacb61bddaa8c46fda0958da7bb002461
Successfully built raymarching
Installing collected packages: raymarching
Successfully installed raymarching-0.0.0
Looking in indexes: https://pypi.org/simple, https://us-python.pkg.dev/colab-wheels/public/simple/
Processing /content/dreamfields-3D/shencoder
Preparing metadata (setup.py) ... done
Building wheels for collected packages: shencoder
Building wheel for shencoder (setup.py) ... done
Created wheel for shencoder: filename=shencoder-0.0.0-cp39-cp39-linux_x86_64.whl size=2628004 sha256=1edd95f5178afd65aa4b8e25bd9071cc3dbf9a531f89b400f1e2197b99cfa2fb
Stored in directory: /tmp/pip-ephem-wheel-cache-mmw2u1d1/wheels/de/27/49/a444fb2e4ffd652437a4f6b66f7af43d054ea49299e17fff1e
Successfully built shencoder
Installing collected packages: shencoder
Successfully installed shencoder-0.0.0
Cloning into 'PyMarchingCubes'...
remote: Enumerating objects: 331, done.
remote: Total 331 (delta 0), reused 0 (delta 0), pack-reused 331
Receiving objects: 100% (331/331), 609.82 KiB | 1.36 MiB/s, done.
Resolving deltas: 100% (184/184), done.
Cloning into 'eigen'...
remote: Enumerating objects: 118903, done.
remote: Counting objects: 100% (582/582), done.
remote: Compressing objects: 100% (209/209), done.
remote: Total 118903 (delta 395), reused 542 (delta 373), pack-reused 118321
Receiving objects: 100% (118903/118903), 103.17 MiB | 35.21 MiB/s, done.
Resolving deltas: 100% (98230/98230), done.
Note: switching to 'tags/3.4.0'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
HEAD is now at 3147391d9 Change version to 3.4.0.
/usr/local/lib/python3.9/dist-packages/setuptools/__init__.py:85: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated. Requirements should be satisfied by a PEP 517 installer. If you are using pip, you can try `pip install --use-pep517`.
dist.fetch_build_eggs(dist.setup_requires)
running install
/usr/local/lib/python3.9/dist-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
Compiling marching_cubes/src/_mcubes.pyx because it changed.
[1/1] Cythonizing marching_cubes/src/_mcubes.pyx
/usr/local/lib/python3.9/dist-packages/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /content/dreamfields-3D/PyMarchingCubes/marching_cubes/src/_mcubes.pyx
tree = Parsing.p_module(s, pxd, full_module_name)
/usr/local/lib/python3.9/dist-packages/setuptools/command/easy_install.py:144: EasyInstallDeprecationWarning: easy_install command is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running bdist_egg
running egg_info
creating PyMarchingCubes.egg-info
writing PyMarchingCubes.egg-info/PKG-INFO
writing dependency_links to PyMarchingCubes.egg-info/dependency_links.txt
writing top-level names to PyMarchingCubes.egg-info/top_level.txt
writing manifest file 'PyMarchingCubes.egg-info/SOURCES.txt'
reading manifest file 'PyMarchingCubes.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'images/smoothing.png'
warning: no files found matching 'README.rst'
adding license file 'LICENSE'
writing manifest file 'PyMarchingCubes.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib.linux-x86_64-3.9
creating build/lib.linux-x86_64-3.9/marching_cubes
copying marching_cubes/smoothing.py -> build/lib.linux-x86_64-3.9/marching_cubes
copying marching_cubes/exporter.py -> build/lib.linux-x86_64-3.9/marching_cubes
copying marching_cubes/__init__.py -> build/lib.linux-x86_64-3.9/marching_cubes
copying marching_cubes/numpy_smoothing.py -> build/lib.linux-x86_64-3.9/marching_cubes
running build_ext
building 'marching_cubes._mcubes' extension
creating build/temp.linux-x86_64-3.9
creating build/temp.linux-x86_64-3.9/marching_cubes
creating build/temp.linux-x86_64-3.9/marching_cubes/src
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Imarching_cubes/src -I/usr/local/lib/python3.9/dist-packages/numpy/core/include -I/usr/include/python3.9 -c marching_cubes/src/_mcubes.cpp -o build/temp.linux-x86_64-3.9/marching_cubes/src/_mcubes.o -std=c++11 -Wall
In file included from /usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1960,
from /usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/arrayobject.h:5,
from marching_cubes/src/_mcubes.cpp:795:
/usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with " \
| ^~~~~~~
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Imarching_cubes/src -I/usr/local/lib/python3.9/dist-packages/numpy/core/include -I/usr/include/python3.9 -c marching_cubes/src/marchingcubes.cpp -o build/temp.linux-x86_64-3.9/marching_cubes/src/marchingcubes.o -std=c++11 -Wall
In file included from marching_cubes/src/marchingcubes.cpp:2:
marching_cubes/src/marchingcubes.h: In function ‘void mc::marching_cubes(const vector3&, const vector3&, int, int, int, formula, double, std::vector<double>&, std::vector<typename vector3::size_type>&)’:
marching_cubes/src/marchingcubes.h:56:11: warning: typedef ‘using coord_type = typename vector3::value_type’ locally defined but not used [-Wunused-local-typedefs]
56 | using coord_type = typename vector3::value_type;
| ^~~~~~~~~~
marching_cubes/src/marchingcubes.h:57:11: warning: typedef ‘using size_type = typename vector3::size_type’ locally defined but not used [-Wunused-local-typedefs]
57 | using size_type = typename vector3::size_type;
| ^~~~~~~~~
In file included from marching_cubes/src/marchingcubes.cpp:2:
marching_cubes/src/marchingcubes.h: In function ‘void mc::marching_cubes_color(const vector3&, const vector3&, int, int, int, formula_sdf, formula_color, double, std::vector<double>&, std::vector<typename vector3::size_type>&)’:
marching_cubes/src/marchingcubes.h:519:11: warning: typedef ‘using coord_type = typename vector3::value_type’ locally defined but not used [-Wunused-local-typedefs]
519 | using coord_type = typename vector3::value_type;
| ^~~~~~~~~~
marching_cubes/src/marchingcubes.h:520:11: warning: typedef ‘using size_type = typename vector3::size_type’ locally defined but not used [-Wunused-local-typedefs]
520 | using size_type = typename vector3::size_type;
| ^~~~~~~~~
marching_cubes/src/marchingcubes.h: In function ‘void mc::marching_cubes_super_sampling(const vector3&, const vector3&, int, int, int, int, int, int, formula, formulaX, formulaY, formulaZ, double, std::vector<double>&, std::vector<typename vector3::size_type>&)’:
marching_cubes/src/marchingcubes.h:800:11: warning: typedef ‘using coord_type = typename vector3::value_type’ locally defined but not used [-Wunused-local-typedefs]
800 | using coord_type = typename vector3::value_type;
| ^~~~~~~~~~
marching_cubes/src/marchingcubes.h:801:11: warning: typedef ‘using size_type = typename vector3::size_type’ locally defined but not used [-Wunused-local-typedefs]
801 | using size_type = typename vector3::size_type;
| ^~~~~~~~~
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Imarching_cubes/src -I/usr/local/lib/python3.9/dist-packages/numpy/core/include -I/usr/include/python3.9 -c marching_cubes/src/pywrapper.cpp -o build/temp.linux-x86_64-3.9/marching_cubes/src/pywrapper.o -std=c++11 -Wall
In file included from /usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/ndarraytypes.h:1960,
from /usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/ndarrayobject.h:12,
from /usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/arrayobject.h:5,
from marching_cubes/src/pyarraymodule.h:11,
from marching_cubes/src/pywrapper.h:6,
from marching_cubes/src/pywrapper.cpp:2:
/usr/local/lib/python3.9/dist-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
17 | #warning "Using deprecated NumPy API, disable it with " \
| ^~~~~~~
In file included from marching_cubes/src/pywrapper.cpp:4:
marching_cubes/src/marchingcubes.h: In function ‘void mc::marching_cubes(const vector3&, const vector3&, int, int, int, formula, double, std::vector<double>&, std::vector<typename vector3::size_type>&)’:
marching_cubes/src/marchingcubes.h:56:11: warning: typedef ‘using coord_type = typename vector3::value_type’ locally defined but not used [-Wunused-local-typedefs]
56 | using coord_type = typename vector3::value_type;
| ^~~~~~~~~~
marching_cubes/src/marchingcubes.h:57:11: warning: typedef ‘using size_type = typename vector3::size_type’ locally defined but not used [-Wunused-local-typedefs]
57 | using size_type = typename vector3::size_type;
| ^~~~~~~~~
In file included from marching_cubes/src/pywrapper.cpp:4:
marching_cubes/src/marchingcubes.h: In function ‘void mc::marching_cubes_color(const vector3&, const vector3&, int, int, int, formula_sdf, formula_color, double, std::vector<double>&, std::vector<typename vector3::size_type>&)’:
marching_cubes/src/marchingcubes.h:519:11: warning: typedef ‘using coord_type = typename vector3::value_type’ locally defined but not used [-Wunused-local-typedefs]
519 | using coord_type = typename vector3::value_type;
| ^~~~~~~~~~
marching_cubes/src/marchingcubes.h:520:11: warning: typedef ‘using size_type = typename vector3::size_type’ locally defined but not used [-Wunused-local-typedefs]
520 | using size_type = typename vector3::size_type;
| ^~~~~~~~~
marching_cubes/src/marchingcubes.h: In function ‘void mc::marching_cubes_super_sampling(const vector3&, const vector3&, int, int, int, int, int, int, formula, formulaX, formulaY, formulaZ, double, std::vector<double>&, std::vector<typename vector3::size_type>&)’:
marching_cubes/src/marchingcubes.h:800:11: warning: typedef ‘using coord_type = typename vector3::value_type’ locally defined but not used [-Wunused-local-typedefs]
800 | using coord_type = typename vector3::value_type;
| ^~~~~~~~~~
marching_cubes/src/marchingcubes.h:801:11: warning: typedef ‘using size_type = typename vector3::size_type’ locally defined but not used [-Wunused-local-typedefs]
801 | using size_type = typename vector3::size_type;
| ^~~~~~~~~
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -Wl,-Bsymbolic-functions -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.9/marching_cubes/src/_mcubes.o build/temp.linux-x86_64-3.9/marching_cubes/src/marchingcubes.o build/temp.linux-x86_64-3.9/marching_cubes/src/pywrapper.o -o build/lib.linux-x86_64-3.9/marching_cubes/_mcubes.cpython-39-x86_64-linux-gnu.so
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/marching_cubes
copying build/lib.linux-x86_64-3.9/marching_cubes/smoothing.py -> build/bdist.linux-x86_64/egg/marching_cubes
copying build/lib.linux-x86_64-3.9/marching_cubes/exporter.py -> build/bdist.linux-x86_64/egg/marching_cubes
copying build/lib.linux-x86_64-3.9/marching_cubes/__init__.py -> build/bdist.linux-x86_64/egg/marching_cubes
copying build/lib.linux-x86_64-3.9/marching_cubes/_mcubes.cpython-39-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/egg/marching_cubes
copying build/lib.linux-x86_64-3.9/marching_cubes/numpy_smoothing.py -> build/bdist.linux-x86_64/egg/marching_cubes
byte-compiling build/bdist.linux-x86_64/egg/marching_cubes/smoothing.py to smoothing.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/marching_cubes/exporter.py to exporter.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/marching_cubes/__init__.py to __init__.cpython-39.pyc
byte-compiling build/bdist.linux-x86_64/egg/marching_cubes/numpy_smoothing.py to numpy_smoothing.cpython-39.pyc
creating stub loader for marching_cubes/_mcubes.cpython-39-x86_64-linux-gnu.so
byte-compiling build/bdist.linux-x86_64/egg/marching_cubes/_mcubes.py to _mcubes.cpython-39.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying PyMarchingCubes.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying PyMarchingCubes.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying PyMarchingCubes.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying PyMarchingCubes.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
marching_cubes.__pycache__._mcubes.cpython-39: module references __file__
creating dist
creating 'dist/PyMarchingCubes-0.0.2-py3.9-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing PyMarchingCubes-0.0.2-py3.9-linux-x86_64.egg
creating /usr/local/lib/python3.9/dist-packages/PyMarchingCubes-0.0.2-py3.9-linux-x86_64.egg
Extracting PyMarchingCubes-0.0.2-py3.9-linux-x86_64.egg to /usr/local/lib/python3.9/dist-packages
Adding PyMarchingCubes 0.0.2 to easy-install.pth file
Installed /usr/local/lib/python3.9/dist-packages/PyMarchingCubes-0.0.2-py3.9-linux-x86_64.egg
Processing dependencies for PyMarchingCubes==0.0.2
Finished processing dependencies for PyMarchingCubes==0.0.2
#@title ###Define necessary functions
#find the path of marching_cubes
import site
import glob
import sys
def find_pm_path(pm_path = '/usr/local/lib/python3.7/dist-packages/'):
pm_path = pm_path + "/PyMarchingCubes*/"
path = sorted(glob.glob(pm_path), reverse=True)
if path:
return path[0]
pm_path = find_pm_path(site.getsitepackages()[0])
# marching_cubes_path = r"/usr/local/lib/python3.7/dist-packages/PyMarchingCubes-0.0.2-py3.7-linux-x86_64.egg" #it depend on your OS but just paste the path where is mcubes
if not pm_path in sys.path:
sys.path.append(pm_path)
import marching_cubes as mcubes
def get_latest_dir(path):
dir_list = os.listdir(path)
dir_list.sort(key=lambda x:os.path.getmtime((path+"/"+x)))
assert len(dir_list) >= 1, "there is not previous project in present output_dir, please check the path"
folder_name = os.path.join(output_dir, dir_list[-1])
folder_name = os.path.basename(folder_name)
return folder_name
def get_latest_file(path):
dir_list = os.listdir(path)
dir_list.sort(key=lambda x:os.path.getmtime((path+"/"+x)))
return dir_list[-1]
Parameters introduction:
# from typing_extensions import Text
#@markdown ####**Training Settings:**
Prompt_text = "KAWS as a Grim Reaper in 8k, High Detail, Skull as a Face" #@param{type: 'string'}
Prompt_image = "" #@param{type: 'string'}
Prompt_image_direction = 'None' #@param ['None','front', 'left side', 'back', 'right side', 'top', 'bottom']
Prompt_image_direction = "" if Prompt_image_direction == 'None' else "--image_direction " + "'" + Prompt_image_direction + "'"
Epoch_num = 500 #@param {type: 'integer'}
learning_rate = 0.0005 #@param {type: 'number'}
dt_gamma = 0 #@param {type: 'number'}
render_W_H = 384 #@param {type: 'integer'}
clip_w_h = 224 #@param {type: 'integer'}
clip_aug_copy = 8 #@param {type: 'integer'}
clip_aug = True #@param {type:"boolean"}
clip_aug = "--clip_aug " if clip_aug else ""
use_clip_direction_text = True #@param {type:"boolean"}
use_clip_direction_text = "--dir_text " if use_clip_direction_text else ""
clip_model = 'ViT-L/14' #@param ['ViT-B/32','ViT-B/16','ViT-L/14','ViT-L/14@336px','RN50x16','RN50x64']
seed = -1 #@param {type: 'integer'}
camera_fovy = 43 #@param {type: 'integer'}
random_fovy_training = True #@param {type:"boolean"}
random_fovy_training = "--rnd_fovy" if random_fovy_training else ""
##@markdown * set seed to -1 to get random seed
resume_train = True #@param {type:"boolean"}
resume_project = "latest" #@param{type: 'string'}
ckpt = "latest" #@param{type: 'string'}
#@markdown ---
#@markdown ####**Output Settings:**
output_dir = "/content/drive/MyDrive/Dreamfields3D_output/" #@param{type: 'string'}
project_name = "grim" #@param{type: 'string'}
display_interval = 10 #@param {type: "integer"}
interval_samples = 12 #@param {type: "integer"}
save_interval_img = False #@param {type:"boolean"}
save_interval_img = "--save_interval_img" if save_interval_img else ""
save_depth = False #@param {type:"boolean"}
save_depth = "--save_depth" if save_depth else ""
ckpt_save_interval = 50 #@param {type: "integer"}
mesh_resolution = 256 #@param {type: "integer"}
mesh_threshold = 10 #@param {type: "integer"}
# assign_subdir = True #@param {type:"boolean"}
#paramater conversion
import time
import os
#convert the prompt text
Prompt_text = "'" + Prompt_text + "'"
iteration = Epoch_num * 100
#get the workspace folder
try:
DC_dir
except NameError:
DC_dir = '/content/dreamfields-3D/'
assert os.path.isfile (DC_dir + '/main_nerf.py'), \
"wrong working folder for dreamfileds3D, please enter the right folder and run again"
if not os.path.exists(output_dir):
os.mkdir(output_dir)
#check the parameters
assert clip_w_h % 8 ==0, f"the value of clip_w_h should be divisible by 8, but current value {clip_w_h} is not, try to use 224 or 128"
#convert the augments
if not resume_train:
workspace = project_name + "_" + time.strftime('%Y%m%d-%H%M%S')[4:-2]
if resume_train:
if resume_project == "latest":
workspace = get_latest_dir(output_dir)
else:
workspace = resume_project
if Prompt_image != "":
Prompt_image = "--image " + "'" + Prompt_image + "'"
if not os.path.exists(output_dir):
os.makedirs(output_dir)
#@title Training
%cd {DC_dir}
%run main_nerf.py --text {Prompt_text} {Prompt_image} {Prompt_image_direction} --cuda_ray --fp16 --iters {iteration} --seed {seed} \
--output_dir {output_dir} --workspace {workspace} --colab --val_int {display_interval} --val_samples {interval_samples} --w {clip_w_h} --h {clip_w_h} \
--H {render_W_H} --W {render_W_H} --fovy {camera_fovy} --clip_model {clip_model} --aug_copy {clip_aug_copy} --lr {learning_rate} --dt_gamma {dt_gamma} \
--ckpt_save_interval {ckpt_save_interval} {save_interval_img} {use_clip_direction_text} {clip_aug} {random_fovy_training} --ckpt {ckpt} {save_depth}
%cd {DC_dir}
++> Evaluate at epoch 300 ...
loss=-0.8714 (-0.8497): : 8% 1/12 [00:00<00:01, 6.50it/s]
[density grid] min=0.0000, max=1.6407, mean=0.0169, occ_rate=0.058 | [step counter] mean=507607
loss=-0.8484 (-0.8826): : 92% 11/12 [00:00<00:00, 13.08it/s]
loss=-0.8484 (-0.8826): : 100% 12/12 [00:01<00:00, 9.66it/s]
++> Evaluate epoch 300 Finished.
[INFO] New best result: -0.8745408107837042 --> -0.8825522661209106
==> Start Training Epoch 301, lr=0.000493 ...
[density grid] min=0.0000, max=1.7186, mean=0.0172, occ_rate=0.058 | [step counter] mean=507607
loss=-0.8084 (-0.8665): : 100% 100/100 [00:11<00:00, 8.82it/s]
==> Finished Epoch 301.
==> Start Training Epoch 302, lr=0.000493 ...
[density grid] min=0.0000, max=1.7947, mean=0.0176, occ_rate=0.058 | [step counter] mean=448596
loss=-0.7118 (-0.8454): : 100% 100/100 [00:11<00:00, 8.78it/s]
==> Finished Epoch 302.
==> Start Training Epoch 303, lr=0.000493 ...
[density grid] min=0.0000, max=1.8556, mean=0.0188, occ_rate=0.058 | [step counter] mean=281461
loss=-0.8119 (-0.8468): : 57% 57/100 [00:06<00:04, 8.69it/s]
Generate image / video / mesh with latest pretrained checkpoint file.
# from typing_extensions import Text
#@markdown ####**Test Settings:**
test_project = "latest" #@param{type: 'string'}
ckpt = "latest" #@param{type: 'string'}
test_samples = 240 #@param {type: "integer"}
render_W_H = 1024 #@param {type: 'integer'}
mesh_resolution = 256 #@param {type: "integer"}
mesh_threshold = 10 #@param {type: "integer"}
save_depth = False #@param {type:"boolean"}
save_depth = "--save_depth" if save_depth else ""
save_video = False #@param {type:"boolean"}
save_video = "--save_video" if save_video else ""
save_mesh = True #@param {type:"boolean"}
save_mesh = "--save_mesh" if save_mesh else ""
#convert the parameters
if test_project == "latest" :
workspace = get_latest_dir(output_dir)
else:
workspace = test_project
#@title Testing
#@ markdown Will generate images and mesh output use the latest checkpoint, even when the training has not finished.
%cd {DC_dir}
%run main_nerf.py --text {Prompt_text} --cuda_ray --fp16 --iters {iteration} --seed {seed} \
--output_dir {output_dir} --workspace {workspace} --colab --val_int {display_interval} --w {clip_w_h} --h {clip_w_h} \
--H {render_W_H} --W {render_W_H} --fovy {camera_fovy} --clip_model {clip_model} --aug_copy {clip_aug_copy} --dt_gamma 0 \
--ckpt_save_interval {ckpt_save_interval} {save_interval_img} {use_clip_direction_text} {clip_aug} --ckpt {ckpt} {save_depth} \
--test_samples {test_samples} --mesh_res {mesh_resolution} --mesh_trh {mesh_threshold} {save_video} {save_mesh} \
--test
%cd {DC_dir}
#@title display latest RGB video
from IPython.display import HTML
from base64 import b64encode
path2video = f"{output_dir}/{workspace}/videos/" + get_latest_file (f"{output_dir}/{workspace}/videos/")
def show_video(video_path, video_width = 600):
video_file = open(video_path, "r+b").read()
video_url = f"data:video/mp4;base64,{b64encode(video_file).decode()}"
return HTML(f"""<video width={video_width} controls><source src="{video_url}"></video>""")
show_video(path2video)